Skip to content

Fix TestNG session status reported with EFD#11087

Draft
daniel-mohedano wants to merge 2 commits intomasterfrom
daniel.mohedano/testng-failure-propagation
Draft

Fix TestNG session status reported with EFD#11087
daniel-mohedano wants to merge 2 commits intomasterfrom
daniel.mohedano/testng-failure-propagation

Conversation

@daniel-mohedano
Copy link
Copy Markdown
Contributor

@daniel-mohedano daniel-mohedano commented Apr 13, 2026 •
edited
Loading

What Does This Do

  • Fixes a bug that would make the test session status dependent of the result orders when using EFD with TestNG:
    • If the result order was pass and fail, the test session was correctly failed.
    • If the result order was fail and pass, the test session would incorrectly pass, while EFD should fail the session when a flaky is found.
  • This was caused by TestNG's retry analyzer suppressing failures on retried tests automatically, which would only trigger if the first test execution was a fail.
  • This bug only affects EFD as it depends on tests being retried after a failure and the failures not being suppressed:
    • ATR suppresses failures by default
    • Attempt to Fix will not retry a test if the first execution is a failure

Motivation

Aligns EFD behavior for TestNG with the rest of the testing frameworks.

Additional Information

Two unit tests have been introduced to verify the behavior.

Contributor Checklist

Jira ticket: SDTEST-3720

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

@daniel-mohedano daniel-mohedano added type: bug Bug report and fix comp: ci visibility Continuous Integration Visibility labels Apr 13, 2026
@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - sbt-scalatest

Job Status: success

Scenario Overhead (%)
agent 56.07
agentEvpProxy 56.71

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - nebula-release-plugin

Job Status: success

Scenario Overhead (%)
agent 37.89
agentless 35.92
agentlessCodeCoverage 44.74
agentlessLineCoverage 75.02

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - netflix-zuul

Job Status: success

Scenario Overhead (%)
agent 85.88
agentless 79.61
agentlessCodeCoverage 95.99
agentlessLineCoverage 112.69

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - reactive-streams-jvm

Job Status: success

Scenario Overhead (%)
agent 22.43
agentless 19.15
agentlessCodeCoverage 20.46
agentlessLineCoverage 31.26

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - pass4s

Job Status: success

Scenario Overhead (%)
agent 8.94
agentless 8.00
agentlessCodeCoverage 16.46

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - sonar-kotlin

Job Status: success

Scenario Overhead (%)
agent 12.63
agentless 11.37
agentlessCodeCoverage 15.47
agentlessLineCoverage 19.22

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - jolokia

Job Status: success

Scenario Overhead (%)
agent 94.05
agentless 88.32
agentlessCodeCoverage 98.62
agentlessLineCoverage 99.63

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - okhttp

Job Status: success

Scenario Overhead (%)
agent 19.83
agentless 19.36
agentlessCodeCoverage 23.75
agentlessLineCoverage 45.30

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - spring_boot

Job Status: success

Scenario Overhead (%)
agent 16.13
agentless 9.55
agentlessCodeCoverage 14.09
agentlessLineCoverage 34.06

@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Apr 13, 2026 •
edited
Loading

Benchmarks

⚠️ Warning: Baseline build not found for merge-base commit. Comparing against the latest commit on master instead.

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/testng-failure-propagation
git_commit_date 1775834061 1776083507
git_commit_sha 5ab378f a38d861
release_version 1.62.0-SNAPSHOT~5ab378f780 1.62.0-SNAPSHOT~a38d861db1
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1776085374 1776085374
ci_job_id 1589299144 1589299144
ci_pipeline_id 107350415 107350415
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-j9zc1dmb 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-j9zc1dmb 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 13 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.62.0-SNAPSHOT~a38d861db1, baseline=1.62.0-SNAPSHOT~5ab378f780

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1056429
Total [baseline] (8.833 s) : 0, 8832858
Agent [candidate] (1.068 s) : 0, 1068181
Total [candidate] (8.878 s) : 0, 8877605
section iast
Agent [baseline] (1.223 s) : 0, 1222800
Total [baseline] (9.551 s) : 0, 9550696
Agent [candidate] (1.233 s) : 0, 1232883
Total [candidate] (9.611 s) : 0, 9611165
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.056 s -
Agent iast 1.223 s 166.371 ms (15.7%)
Total tracing 8.833 s -
Total iast 9.551 s 717.838 ms (8.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.068 s -
Agent iast 1.233 s 164.702 ms (15.4%)
Total tracing 8.878 s -
Total iast 9.611 s 733.56 ms (8.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.62.0-SNAPSHOT~a38d861db1, baseline=1.62.0-SNAPSHOT~5ab378f780

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.227 ms) : 0, 1227
crashtracking [candidate] (1.238 ms) : 0, 1238
BytebuddyAgent [baseline] (632.183 ms) : 0, 632183
BytebuddyAgent [candidate] (639.143 ms) : 0, 639143
AgentMeter [baseline] (29.494 ms) : 0, 29494
AgentMeter [candidate] (29.741 ms) : 0, 29741
GlobalTracer [baseline] (248.37 ms) : 0, 248370
GlobalTracer [candidate] (250.747 ms) : 0, 250747
AppSec [baseline] (31.938 ms) : 0, 31938
AppSec [candidate] (32.345 ms) : 0, 32345
Debugger [baseline] (59.255 ms) : 0, 59255
Debugger [candidate] (59.834 ms) : 0, 59834
Remote Config [baseline] (594.844 µs) : 0, 595
Remote Config [candidate] (605.979 µs) : 0, 606
Telemetry [baseline] (8.096 ms) : 0, 8096
Telemetry [candidate] (8.247 ms) : 0, 8247
Flare Poller [baseline] (9.144 ms) : 0, 9144
Flare Poller [candidate] (9.879 ms) : 0, 9879
section iast
crashtracking [baseline] (1.227 ms) : 0, 1227
crashtracking [candidate] (1.231 ms) : 0, 1231
BytebuddyAgent [baseline] (800.446 ms) : 0, 800446
BytebuddyAgent [candidate] (806.619 ms) : 0, 806619
AgentMeter [baseline] (11.379 ms) : 0, 11379
AgentMeter [candidate] (11.498 ms) : 0, 11498
GlobalTracer [baseline] (238.906 ms) : 0, 238906
GlobalTracer [candidate] (241.698 ms) : 0, 241698
IAST [baseline] (25.813 ms) : 0, 25813
IAST [candidate] (26.145 ms) : 0, 26145
AppSec [baseline] (31.072 ms) : 0, 31072
AppSec [candidate] (32.149 ms) : 0, 32149
Debugger [baseline] (60.837 ms) : 0, 60837
Debugger [candidate] (59.556 ms) : 0, 59556
Remote Config [baseline] (533.806 µs) : 0, 534
Remote Config [candidate] (542.29 µs) : 0, 542
Telemetry [baseline] (12.359 ms) : 0, 12359
Telemetry [candidate] (13.22 ms) : 0, 13220
Flare Poller [baseline] (3.446 ms) : 0, 3446
Flare Poller [candidate] (3.538 ms) : 0, 3538
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.62.0-SNAPSHOT~a38d861db1, baseline=1.62.0-SNAPSHOT~5ab378f780

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.06 s) : 0, 1060163
Total [baseline] (11.007 s) : 0, 11007148
Agent [candidate] (1.067 s) : 0, 1066806
Total [candidate] (11.113 s) : 0, 11112820
section appsec
Agent [baseline] (1.255 s) : 0, 1254594
Total [baseline] (11.192 s) : 0, 11192125
Agent [candidate] (1.264 s) : 0, 1263714
Total [candidate] (11.368 s) : 0, 11368177
section iast
Agent [baseline] (1.225 s) : 0, 1224918
Total [baseline] (11.406 s) : 0, 11406388
Agent [candidate] (1.227 s) : 0, 1227046
Total [candidate] (11.424 s) : 0, 11423893
section profiling
Agent [baseline] (1.186 s) : 0, 1185867
Total [baseline] (11.074 s) : 0, 11073598
Agent [candidate] (1.19 s) : 0, 1189766
Total [candidate] (11.289 s) : 0, 11289306
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.06 s -
Agent appsec 1.255 s 194.431 ms (18.3%)
Agent iast 1.225 s 164.755 ms (15.5%)
Agent profiling 1.186 s 125.705 ms (11.9%)
Total tracing 11.007 s -
Total appsec 11.192 s 184.977 ms (1.7%)
Total iast 11.406 s 399.24 ms (3.6%)
Total profiling 11.074 s 66.45 ms (0.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.067 s -
Agent appsec 1.264 s 196.908 ms (18.5%)
Agent iast 1.227 s 160.239 ms (15.0%)
Agent profiling 1.19 s 122.959 ms (11.5%)
Total tracing 11.113 s -
Total appsec 11.368 s 255.358 ms (2.3%)
Total iast 11.424 s 311.073 ms (2.8%)
Total profiling 11.289 s 176.486 ms (1.6%) 
gantt
    title petclinic - break down per module: candidate=1.62.0-SNAPSHOT~a38d861db1, baseline=1.62.0-SNAPSHOT~5ab378f780

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.242 ms) : 0, 1242
crashtracking [candidate] (1.239 ms) : 0, 1239
BytebuddyAgent [baseline] (634.102 ms) : 0, 634102
BytebuddyAgent [candidate] (637.476 ms) : 0, 637476
AgentMeter [baseline] (29.397 ms) : 0, 29397
AgentMeter [candidate] (29.761 ms) : 0, 29761
GlobalTracer [baseline] (248.958 ms) : 0, 248958
GlobalTracer [candidate] (250.434 ms) : 0, 250434
AppSec [baseline] (32.1 ms) : 0, 32100
AppSec [candidate] (32.204 ms) : 0, 32204
Debugger [baseline] (60.134 ms) : 0, 60134
Debugger [candidate] (60.17 ms) : 0, 60170
Remote Config [baseline] (607.76 µs) : 0, 608
Remote Config [candidate] (607.255 µs) : 0, 607
Telemetry [baseline] (8.153 ms) : 0, 8153
Telemetry [candidate] (8.089 ms) : 0, 8089
Flare Poller [baseline] (9.186 ms) : 0, 9186
Flare Poller [candidate] (10.534 ms) : 0, 10534
section appsec
crashtracking [baseline] (1.239 ms) : 0, 1239
crashtracking [candidate] (1.233 ms) : 0, 1233
BytebuddyAgent [baseline] (665.622 ms) : 0, 665622
BytebuddyAgent [candidate] (668.459 ms) : 0, 668459
AgentMeter [baseline] (12.117 ms) : 0, 12117
AgentMeter [candidate] (12.3 ms) : 0, 12300
GlobalTracer [baseline] (250.032 ms) : 0, 250032
GlobalTracer [candidate] (253.488 ms) : 0, 253488
IAST [baseline] (24.776 ms) : 0, 24776
IAST [candidate] (25.027 ms) : 0, 25027
AppSec [baseline] (185.408 ms) : 0, 185408
AppSec [candidate] (186.642 ms) : 0, 186642
Debugger [baseline] (66.101 ms) : 0, 66101
Debugger [candidate] (66.86 ms) : 0, 66860
Remote Config [baseline] (611.993 µs) : 0, 612
Remote Config [candidate] (635.596 µs) : 0, 636
Telemetry [baseline] (8.664 ms) : 0, 8664
Telemetry [candidate] (8.801 ms) : 0, 8801
Flare Poller [baseline] (3.572 ms) : 0, 3572
Flare Poller [candidate] (3.692 ms) : 0, 3692
section iast
crashtracking [baseline] (1.223 ms) : 0, 1223
crashtracking [candidate] (1.231 ms) : 0, 1231
BytebuddyAgent [baseline] (801.891 ms) : 0, 801891
BytebuddyAgent [candidate] (801.871 ms) : 0, 801871
AgentMeter [baseline] (11.401 ms) : 0, 11401
AgentMeter [candidate] (11.494 ms) : 0, 11494
GlobalTracer [baseline] (239.052 ms) : 0, 239052
GlobalTracer [candidate] (240.333 ms) : 0, 240333
IAST [baseline] (26.715 ms) : 0, 26715
IAST [candidate] (25.974 ms) : 0, 25974
AppSec [baseline] (31.797 ms) : 0, 31797
AppSec [candidate] (30.453 ms) : 0, 30453
Debugger [baseline] (59.588 ms) : 0, 59588
Debugger [candidate] (61.912 ms) : 0, 61912
Remote Config [baseline] (532.351 µs) : 0, 532
Remote Config [candidate] (1.74 ms) : 0, 1740
Telemetry [baseline] (12.887 ms) : 0, 12887
Telemetry [candidate] (12.192 ms) : 0, 12192
Flare Poller [baseline] (3.493 ms) : 0, 3493
Flare Poller [candidate] (3.636 ms) : 0, 3636
section profiling
crashtracking [baseline] (1.177 ms) : 0, 1177
crashtracking [candidate] (1.19 ms) : 0, 1190
BytebuddyAgent [baseline] (692.089 ms) : 0, 692089
BytebuddyAgent [candidate] (694.444 ms) : 0, 694444
AgentMeter [baseline] (9.17 ms) : 0, 9170
AgentMeter [candidate] (9.122 ms) : 0, 9122
GlobalTracer [baseline] (207.142 ms) : 0, 207142
GlobalTracer [candidate] (207.35 ms) : 0, 207350
AppSec [baseline] (32.584 ms) : 0, 32584
AppSec [candidate] (32.741 ms) : 0, 32741
Debugger [baseline] (65.773 ms) : 0, 65773
Debugger [candidate] (65.876 ms) : 0, 65876
Remote Config [baseline] (581.408 µs) : 0, 581
Remote Config [candidate] (578.076 µs) : 0, 578
Telemetry [baseline] (7.862 ms) : 0, 7862
Telemetry [candidate] (7.93 ms) : 0, 7930
Flare Poller [baseline] (3.61 ms) : 0, 3610
Flare Poller [candidate] (3.623 ms) : 0, 3623
ProfilingAgent [baseline] (94.55 ms) : 0, 94550
ProfilingAgent [candidate] (95.248 ms) : 0, 95248
Profiling [baseline] (95.123 ms) : 0, 95123
Profiling [candidate] (95.825 ms) : 0, 95825
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/testng-failure-propagation
git_commit_date 1775834061 1776083507
git_commit_sha 5ab378f a38d861
release_version 1.62.0-SNAPSHOT~5ab378f780 1.62.0-SNAPSHOT~a38d861db1
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1776085879 1776085879
ci_job_id 1589299148 1589299148
ci_pipeline_id 107350415 107350415
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-kb06nuc5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-kb06nuc5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 2 performance regressions! Performance is the same for 17 metrics, 17 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:iast_GLOBAL:high_load worse
[+167.966µs; +325.678µs] or [+6.279%; +12.174%]		 worse
[+177.015µs; +689.356µs] or [+2.305%; +8.978%]		 unstable
[-228.309op/s; +69.684op/s] or [-17.133%; +5.229%]		 2.922ms 8.111ms 1253.250op/s 2.675ms 7.678ms 1332.562op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~a38d861db1, baseline=1.62.0-SNAPSHOT~5ab378f780
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.136 ms) : 17953, 18320
.   : milestone, 18136,
appsec (18.525 ms) : 18336, 18714
.   : milestone, 18525,
code_origins (18.022 ms) : 17845, 18200
.   : milestone, 18022,
iast (17.551 ms) : 17379, 17722
.   : milestone, 17551,
profiling (18.312 ms) : 18130, 18494
.   : milestone, 18312,
tracing (18.063 ms) : 17884, 18242
.   : milestone, 18063,
section candidate
no_agent (18.121 ms) : 17934, 18308
.   : milestone, 18121,
appsec (18.585 ms) : 18391, 18780
.   : milestone, 18585,
code_origins (18.119 ms) : 17938, 18299
.   : milestone, 18119,
iast (18.149 ms) : 17967, 18330
.   : milestone, 18149,
profiling (18.489 ms) : 18308, 18670
.   : milestone, 18489,
tracing (17.855 ms) : 17679, 18031
.   : milestone, 17855,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.136 ms [17.953 ms, 18.32 ms] -
appsec 18.525 ms [18.336 ms, 18.714 ms] 388.544 µs (2.1%)
code_origins 18.022 ms [17.845 ms, 18.2 ms] -114.212 µs (-0.6%)
iast 17.551 ms [17.379 ms, 17.722 ms] -585.715 µs (-3.2%)
profiling 18.312 ms [18.13 ms, 18.494 ms] 175.728 µs (1.0%)
tracing 18.063 ms [17.884 ms, 18.242 ms] -73.29 µs (-0.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.121 ms [17.934 ms, 18.308 ms] -
appsec 18.585 ms [18.391 ms, 18.78 ms] 464.252 µs (2.6%)
code_origins 18.119 ms [17.938 ms, 18.299 ms] -2.418 µs (-0.0%)
iast 18.149 ms [17.967 ms, 18.33 ms] 27.391 µs (0.2%)
profiling 18.489 ms [18.308 ms, 18.67 ms] 368.03 µs (2.0%)
tracing 17.855 ms [17.679 ms, 18.031 ms] -266.237 µs (-1.5%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.62.0-SNAPSHOT~a38d861db1, baseline=1.62.0-SNAPSHOT~5ab378f780
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.239 ms) : 1226, 1251
.   : milestone, 1239,
iast (3.307 ms) : 3260, 3354
.   : milestone, 3307,
iast_FULL (6.093 ms) : 6031, 6155
.   : milestone, 6093,
iast_GLOBAL (3.437 ms) : 3382, 3493
.   : milestone, 3437,
profiling (1.973 ms) : 1955, 1992
.   : milestone, 1973,
tracing (1.942 ms) : 1926, 1958
.   : milestone, 1942,
section candidate
no_agent (1.256 ms) : 1244, 1268
.   : milestone, 1256,
iast (3.282 ms) : 3236, 3327
.   : milestone, 3282,
iast_FULL (5.935 ms) : 5875, 5995
.   : milestone, 5935,
iast_GLOBAL (3.661 ms) : 3603, 3719
.   : milestone, 3661,
profiling (2.142 ms) : 2123, 2161
.   : milestone, 2142,
tracing (1.885 ms) : 1869, 1901
.   : milestone, 1885,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.239 ms [1.226 ms, 1.251 ms] -
iast 3.307 ms [3.26 ms, 3.354 ms] 2.068 ms (167.0%)
iast_FULL 6.093 ms [6.031 ms, 6.155 ms] 4.854 ms (391.9%)
iast_GLOBAL 3.437 ms [3.382 ms, 3.493 ms] 2.199 ms (177.5%)
profiling 1.973 ms [1.955 ms, 1.992 ms] 734.81 µs (59.3%)
tracing 1.942 ms [1.926 ms, 1.958 ms] 703.186 µs (56.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.256 ms [1.244 ms, 1.268 ms] -
iast 3.282 ms [3.236 ms, 3.327 ms] 2.026 ms (161.2%)
iast_FULL 5.935 ms [5.875 ms, 5.995 ms] 4.679 ms (372.5%)
iast_GLOBAL 3.661 ms [3.603 ms, 3.719 ms] 2.405 ms (191.4%)
profiling 2.142 ms [2.123 ms, 2.161 ms] 885.854 µs (70.5%)
tracing 1.885 ms [1.869 ms, 1.901 ms] 628.499 µs (50.0%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/testng-failure-propagation
git_commit_date 1775834061 1776083507
git_commit_sha 5ab378f a38d861
release_version 1.62.0-SNAPSHOT~5ab378f780 1.62.0-SNAPSHOT~a38d861db1
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1776085455 1776085455
ci_job_id 1589299152 1589299152
ci_pipeline_id 107350415 107350415
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-z71neegp 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-z71neegp 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~a38d861db1, baseline=1.62.0-SNAPSHOT~5ab378f780
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.496 ms) : 1484, 1507
.   : milestone, 1496,
appsec (3.839 ms) : 3616, 4062
.   : milestone, 3839,
iast (2.298 ms) : 2228, 2368
.   : milestone, 2298,
iast_GLOBAL (2.34 ms) : 2270, 2410
.   : milestone, 2340,
profiling (2.106 ms) : 2050, 2161
.   : milestone, 2106,
tracing (2.095 ms) : 2041, 2150
.   : milestone, 2095,
section candidate
no_agent (1.492 ms) : 1481, 1504
.   : milestone, 1492,
appsec (3.845 ms) : 3624, 4066
.   : milestone, 3845,
iast (2.289 ms) : 2219, 2358
.   : milestone, 2289,
iast_GLOBAL (2.327 ms) : 2257, 2398
.   : milestone, 2327,
profiling (2.12 ms) : 2064, 2175
.   : milestone, 2120,
tracing (2.107 ms) : 2052, 2162
.   : milestone, 2107,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.496 ms [1.484 ms, 1.507 ms] -
appsec 3.839 ms [3.616 ms, 4.062 ms] 2.343 ms (156.7%)
iast 2.298 ms [2.228 ms, 2.368 ms] 802.275 µs (53.6%)
iast_GLOBAL 2.34 ms [2.27 ms, 2.41 ms] 844.288 µs (56.4%)
profiling 2.106 ms [2.05 ms, 2.161 ms] 610.037 µs (40.8%)
tracing 2.095 ms [2.041 ms, 2.15 ms] 599.64 µs (40.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.492 ms [1.481 ms, 1.504 ms] -
appsec 3.845 ms [3.624 ms, 4.066 ms] 2.353 ms (157.7%)
iast 2.289 ms [2.219 ms, 2.358 ms] 796.411 µs (53.4%)
iast_GLOBAL 2.327 ms [2.257 ms, 2.398 ms] 835.026 µs (56.0%)
profiling 2.12 ms [2.064 ms, 2.175 ms] 627.325 µs (42.0%)
tracing 2.107 ms [2.052 ms, 2.162 ms] 614.956 µs (41.2%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.62.0-SNAPSHOT~a38d861db1, baseline=1.62.0-SNAPSHOT~5ab378f780
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.969 s) : 14969000, 14969000
.   : milestone, 14969000,
appsec (15.019 s) : 15019000, 15019000
.   : milestone, 15019000,
iast (18.084 s) : 18084000, 18084000
.   : milestone, 18084000,
iast_GLOBAL (18.028 s) : 18028000, 18028000
.   : milestone, 18028000,
profiling (15.391 s) : 15391000, 15391000
.   : milestone, 15391000,
tracing (14.994 s) : 14994000, 14994000
.   : milestone, 14994000,
section candidate
no_agent (15.015 s) : 15015000, 15015000
.   : milestone, 15015000,
appsec (14.91 s) : 14910000, 14910000
.   : milestone, 14910000,
iast (18.439 s) : 18439000, 18439000
.   : milestone, 18439000,
iast_GLOBAL (18.367 s) : 18367000, 18367000
.   : milestone, 18367000,
profiling (15.615 s) : 15615000, 15615000
.   : milestone, 15615000,
tracing (15.105 s) : 15105000, 15105000
.   : milestone, 15105000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.969 s [14.969 s, 14.969 s] -
appsec 15.019 s [15.019 s, 15.019 s] 50.0 ms (0.3%)
iast 18.084 s [18.084 s, 18.084 s] 3.115 s (20.8%)
iast_GLOBAL 18.028 s [18.028 s, 18.028 s] 3.059 s (20.4%)
profiling 15.391 s [15.391 s, 15.391 s] 422.0 ms (2.8%)
tracing 14.994 s [14.994 s, 14.994 s] 25.0 ms (0.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.015 s [15.015 s, 15.015 s] -
appsec 14.91 s [14.91 s, 14.91 s] -105.0 ms (-0.7%)
iast 18.439 s [18.439 s, 18.439 s] 3.424 s (22.8%)
iast_GLOBAL 18.367 s [18.367 s, 18.367 s] 3.352 s (22.3%)
profiling 15.615 s [15.615 s, 15.615 s] 600.0 ms (4.0%)
tracing 15.105 s [15.105 s, 15.105 s] 90.0 ms (0.6%)

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da bot commented Apr 13, 2026 •
edited
Loading

Test Environment - sonar-java

Job Status: success

Scenario Overhead (%)
agent -5.11
agentless -9.61
agentlessCodeCoverage 54.76
agentlessLineCoverage 119.55

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility type: bug Bug report and fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@daniel-mohedano